Replication by ritch · Pull Request #153 · strongloop/loopback

ritch · 2014-01-26T21:28:27Z

Please read this document for background on our approach to replication.

http://docs.strongloop.com/display/DOC/Replication

This includes two new models

Change
Checkpoint

bajtos · 2014-01-27T14:23:52Z

As a developer consuming the API, I would expect Change.track to run continuously. Perhaps Change.pullUpdatesOf is a better name?

Maybe the description needs to be reworded. This method rectifies change list entries for the given model ids. I'm not sure pullUpdatesOf is any less confusing.

Few alternate names based on your description:

Change.rectify (similar to Change.prototype.rectify)

Change.rectifyChangesOfModel

Change.rectifyModelChanges

bajtos · 2014-01-27T14:55:23Z

The code looks mostly good. It's difficult to review it from the big picture point of view, as there are no test cases covering larger and/or integration scenarios.

Perhaps it's better to partition the implementation vertically, start with a full end-to-end implementation of something minimal (e.g. sync a newly created model instance) and expand horizontally later (i.e. add support for deletes, updates, conflicts - each of those can be a single self-contained change/PR).

slnode · 2014-01-28T20:10:19Z

Test FAILed. To trigger a build add comment - ".test\W+please"
Refer to this link for build results: http://ci.strongloop.com/job/loopback/850/

ritch · 2014-01-28T22:47:56Z

@bajtos

Perhaps it's better to partition the implementation vertically, start with a full end-to-end implementation of something minimal (e.g. sync a newly created model instance)

See https://github.com/strongloop/loopback/pull/153/files#diff-e43576aabfd6aa741af8ac58f1b4a699R673 for an example of this.

ritch · 2014-01-28T22:48:05Z

test please

slnode · 2014-01-29T17:22:52Z

Test FAILed. To trigger a build add comment - ".test\W+please"
Refer to this link for build results: http://ci.strongloop.com/job/loopback/852/

slnode · 2014-01-29T18:08:01Z

Test FAILed. To trigger a build add comment - ".test\W+please"
Refer to this link for build results: http://ci.strongloop.com/job/loopback/853/

rmg · 2014-01-29T18:08:28Z

test please

slnode · 2014-01-29T18:41:30Z

Test FAILed. To trigger a build add comment - ".test\W+please"
Refer to this link for build results: http://ci.strongloop.com/job/loopback/854/

ritch · 2014-01-30T22:23:11Z

test please

slnode · 2014-01-30T22:25:36Z

Test PASSed. To trigger a build add comment - ".test\W+please"
Refer to this link for build results: http://ci.strongloop.com/job/loopback/868/

slnode · 2014-01-30T22:49:05Z

Test PASSed. To trigger a build add comment - ".test\W+please"
Refer to this link for build results: http://ci.strongloop.com/job/loopback/870/

slnode · 2014-02-05T23:47:38Z

Test PASSed. To trigger a build add comment - ".test\W+please"
Refer to this link for build results: http://ci.strongloop.com/job/loopback/895/

slnode · 2014-02-06T00:10:51Z

Test PASSed. To trigger a build add comment - ".test\W+please"
Refer to this link for build results: http://ci.strongloop.com/job/loopback/897/

bajtos · 2014-02-07T09:45:14Z

The example does not work:

$ node app.js
createSomeInitialSourceData
replicateSourceToTarget

current SOURCE data
 - red
 - blue
 - green


current TARGET data

updateSomeTargetData

TypeError: Cannot set property 'name' of null
    at /Users/bajtos/src/loopback/loopback/example/replication/app.js:88:16
    at Function.<anonymous> (/Users/bajtos/src/loopback/loopback/node_modules/loopback-datasource-juggler/lib/dao.js:323:5)
    at Memory.<anonymous> (/Users/bajtos/src/loopback/loopback/node_modules/loopback-datasource-juggler/lib/connectors/memory.js:107:5)
    at process._tickCallback (node.js:415:13)

bajtos · 2014-02-07T10:02:13Z

The fact there is only one Change instance per each model instance (id) looks suspicious to me.

Will the current implementation work correctly when there are multiple replication targets, each replicating at different intervals? Consider the following timeline:

Color, Color2, Color3 - set as in example app (Color3 is setup in the same way as Color2)

0. create Color "red"
1. initial replication: Color->Color2, Color->Color3
   - a Change "color-1" of type CREATE with rev=R1 is created
2. make a change to the Color "red"
3. replicate Color -> Color2, keep Color3 outdated
   - the Change "color-1" is updated to UPDATE prev=R1 rev=R2
4. make another change to the Color "red"
5. replicate Color -> Color3
  - the Change "color-1" is updated to UPDATE prev=R2 rev=R3
  - Color3 "red" is based on rev R1, thus we have a conflict?

Am I missing something?

bajtos · 2014-02-07T14:11:05Z

What kind of value is expected here? Is it Change.id, Model.id or something else?

I found it - Model.getSourceId. Could you please mention that in the property description?

…into feature/replication

slnode · 2014-05-16T00:58:42Z

Test Failed. To trigger a build add comment - ".test\W+please"
Refer to this link for build results: http://ci.strongloop.com/job/loopback/1200/

slnode · 2014-05-16T00:59:21Z

Test Failed. To trigger a build add comment - ".test\W+please"
Refer to this link for build results: http://ci.strongloop.com/job/loopback/1202/

slnode · 2014-05-16T17:17:10Z

Test Failed. To trigger a build add comment - ".test\W+please"
Refer to this link for build results: http://ci.strongloop.com/job/loopback/1218/

slnode · 2014-05-16T19:51:39Z

Test Failed. To trigger a build add comment - ".test\W+please"
Refer to this link for build results: http://ci.strongloop.com/job/loopback/1224/

creating a cache - Use the SharedClass class to build the remote connector - Change default base model from Model to DataModel - Fix DataModel errors not logging correct method names - Use the strong-remoting 1.4 resolver API to resolve dynamic remote methods (relation api) - Remove use of fn object for storing remoting meta data

slnode · 2014-05-19T23:14:22Z

Test Failed. To trigger a build add comment - ".test\W+please"
Refer to this link for build results: http://ci.strongloop.com/job/loopback/1237/

slnode · 2014-05-20T03:16:38Z

Test Failed. To trigger a build add comment - ".test\W+please"
Refer to this link for build results: http://ci.strongloop.com/job/loopback/1239/

slnode · 2014-05-20T19:33:04Z

Test Failed. To trigger a build add comment - ".test\W+please"
Refer to this link for build results: http://ci.strongloop.com/job/loopback/1242/

slnode · 2014-05-20T20:07:01Z

Test Failed. To trigger a build add comment - ".test\W+please"
Refer to this link for build results: http://ci.strongloop.com/job/loopback/1244/

Allow juggler to mix in these methods.

slnode · 2014-05-20T21:16:06Z

Test Failed. To trigger a build add comment - ".test\W+please"
Refer to this link for build results: http://ci.strongloop.com/job/loopback/1247/

slnode · 2014-05-20T22:31:33Z

Test Failed. To trigger a build add comment - ".test\W+please"
Refer to this link for build results: http://ci.strongloop.com/job/loopback/1250/

slnode · 2014-05-20T23:36:30Z

Test Failed. To trigger a build add comment - ".test\W+please"
Refer to this link for build results: http://ci.strongloop.com/job/loopback/1255/

bajtos · 2014-06-05T08:03:15Z

@ritch what's the status of this pull request? was it already landed on the 2.0 branch?

ritch · 2014-06-05T14:39:50Z

Yes. Closing.

ritch added 2 commits January 26, 2014 13:20

Add Change model

4be3395

Add Checkpoint model and Model replication methods

1a13a8d

bajtos reviewed Jan 27, 2014
View reviewed changes

ritch added 2 commits January 28, 2014 12:54

Add replication example

2582c3f

Add model tests

a0e595d

ritch added 4 commits January 30, 2014 14:20

Add Change model

cc49d67

Add Checkpoint model and Model replication methods

e3d8005

Add replication example

ab8d1ae

Add model tests

13f6738

ritch added 2 commits February 5, 2014 15:27

fixup! rename Change.track => rectifyModelChanges

0bf7af1

fixup! Assert model exists

867e3ca

bajtos reviewed Feb 7, 2014
View reviewed changes

ritch added 3 commits May 15, 2014 17:27

Rework replication test

d875c51

Rework replication test

2f21f4e

Merge branch 'feature/replication' of github.com:strongloop/loopback …

d58df51

…into feature/replication

Add test for conflicts where both deleted

4d5e788

In progress: rework remoting meta-data

558ea60

Rework change conflict detection

2de33d4

Remove un-rectify-able changes

52eb72d

Ensure changes are created in sync

77bd77e

ritch added 2 commits May 20, 2014 13:45

!fixup Mark DAO methods as delegate

1a8ba60

Allow juggler to mix in these methods.

Depend on juggler@1.6.0

0c925f7

Merge latest from master

d237ae5

This was referenced May 21, 2014

1.9 #272

Closed

2.0 #276

Merged

ritch closed this Jun 5, 2014

superkhau deleted the feature/replication branch September 30, 2016 02:02

superkhau restored the feature/replication branch September 30, 2016 02:03

Uh oh!

Conversation

ritch commented Jan 26, 2014

Uh oh!

bajtos Jan 27, 2014

Choose a reason for hiding this comment

Uh oh!

ritch Jan 28, 2014

Choose a reason for hiding this comment

Uh oh!

bajtos Jan 31, 2014

Choose a reason for hiding this comment

Uh oh!

bajtos commented Jan 27, 2014

Uh oh!

slnode commented Jan 28, 2014

Uh oh!

ritch commented Jan 28, 2014

Uh oh!

ritch commented Jan 28, 2014

Uh oh!

slnode commented Jan 29, 2014

Uh oh!

slnode commented Jan 29, 2014

Uh oh!

rmg commented Jan 29, 2014

Uh oh!

slnode commented Jan 29, 2014

Uh oh!

ritch commented Jan 30, 2014

Uh oh!

slnode commented Jan 30, 2014

Uh oh!

slnode commented Jan 30, 2014

Uh oh!

slnode commented Feb 5, 2014

Uh oh!

slnode commented Feb 6, 2014

Uh oh!

bajtos commented Feb 7, 2014

Uh oh!

bajtos commented Feb 7, 2014

Uh oh!

bajtos Feb 7, 2014

Choose a reason for hiding this comment

Uh oh!

bajtos Feb 7, 2014

Choose a reason for hiding this comment

Uh oh!

slnode commented May 16, 2014

Uh oh!

slnode commented May 16, 2014

Uh oh!

slnode commented May 16, 2014

Uh oh!

slnode commented May 16, 2014

Uh oh!

slnode commented May 19, 2014

Uh oh!

slnode commented May 20, 2014

Uh oh!

slnode commented May 20, 2014

Uh oh!

slnode commented May 20, 2014

Uh oh!

slnode commented May 20, 2014

Uh oh!

slnode commented May 20, 2014

Uh oh!

slnode commented May 20, 2014

Uh oh!

bajtos commented Jun 5, 2014

Uh oh!

ritch commented Jun 5, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone